Troubleshooting the integrated DBCS Automatic Backup

Troubleshooting the integrated Oracle DBCS (Database Cloud Service) Automatic Backup involves identifying and resolving issues related to the backup process. Here are some steps to troubleshoot common problems – Checking Backup Configuration, Storage Availability, Permissions, Connectivity, etc

To determine the root cause of job failures and whether the RMAN backup job works, the first step is to look at the RMAN log files. All RMAN log files are accessible by the user (‘oracle’) and can be found on the host under the directory: /opt/oracle/dcs/log/<SID>/rman/bkup/<oracle_unqname>

Jump to the Oracle user:

$ cd /opt/oracle/dcs/log/<SID>/rman/bkup/<oracle_unqname>

Scroll down to the related logfile of the date:

[oracle@jj2 jj2_phx153]$ pwd
/opt/oracle/dcs/log/jj2/rman/bkup/jj2_phx153
[oracle@jj2 jj2_phx153]$ ls -ltr
total 24
drwxr-xr-x 13 oracle oinstall 4096 Mar 12 03:20 rman_configure
drwxr-xr-x 13 oracle oinstall 4096 Mar 12 03:20 show_rman_archive_log_deletion_policy
drwxr-xr-x 13 oracle oinstall 4096 Mar 12 03:20 rman_crosscheck
drwxr-xr-x 13 oracle oinstall 4096 Mar 12 03:21 rman_backup
drwxr-xr-x 13 oracle oinstall 4096 Mar 12 03:22 rman_list_backup_detail
drwxr-xr-x  3 oracle oinstall 4096 Mar 12 17:32 rman_restore
[oracle@jj2 jj2_phx153]$ cd rman_backup
[oracle@jj2 rman_backup]$ ls -ltr
total 44
drwxr-xr-x 2 oracle oinstall 4096 Mar  2 03:23 2018-03-01
drwxr-xr-x 2 oracle oinstall 4096 Mar  3 03:25 2018-03-02
drwxr-xr-x 2 oracle oinstall 4096 Mar  4 03:26 2018-03-03
drwxr-xr-x 2 oracle oinstall 4096 Mar  5 03:36 2018-03-04
drwxr-xr-x 2 oracle oinstall 4096 Mar  6 03:25 2018-03-05
drwxr-xr-x 2 oracle oinstall 4096 Mar  7 03:24 2018-03-07
drwxr-xr-x 2 oracle oinstall 4096 Mar  8 03:25 2018-03-08
drwxr-xr-x 2 oracle oinstall 4096 Mar  9 03:37 2018-03-09
drwxr-xr-x 2 oracle oinstall 4096 Mar 10 03:21 2018-03-10
drwxr-xr-x 2 oracle oinstall 4096 Mar 11 03:22 2018-03-11
drwxr-xr-x 2 oracle oinstall 4096 Mar 12 03:23 2018-03-12
[oracle@jj2 rman_backup]$ cd 2018-03-12
[oracle@jj2 2018-03-12]$ ls -tlr
total 20
-rw-r--r-- 1 oracle oinstall 14401 Mar 12 03:22 rman_backup_2018-03-12_03-21-24.0169.log
-rw-r--r-- 1 oracle oinstall  1493 Mar 12 03:23 rman_backup_2018-03-12_03-23-14.0230.log
[oracle@jj2 2018-03-12]$

The above step is pure with the command line. Similarly, we can also review the job status using dbcli. We have various attributes associated with dbcli, few examples

(as opc user) $ sudo su -
(as root user) #dbcli list-databases
(as root user) #dbcli list-backupconfigs
(as root user) #dbcli list-jobs | grep -i failure
(as root user) #dbcli describe-job -i <job id> <===job id for the stuck job

Let’s explore the commands to review the outputs

[root@jj2 ~]# dbcli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID     
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
de352ede-ddf0-41a6-9627-29acc90c00c6     jj2        Si       12.1.0.2             true       OLTP     odb2     ASM        Configured   c06d7bfb-813d-4b52-9eeb-7f2ef1d3aa16
[root@jj2 ~]# dbcli list-backupconfigs

ID                                       Name                 RecoveryWindow   CrosscheckEnabled   BackupDestination
---------------------------------------- -------------------- ---------------- ------------------- --------------------
41f6d03e-094d-45ee-aae6-ee9a1648bdd6     bgeyX3bid7o8KMG43gJH_BC 30               true                ObjectStore
[root@jj2 ~]# dbcli list-jobs | grep -i failure
ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
524cb137-76ef-4806-855f-5b2ec6a74bd6     Create recovery-pitr : time '03/11/2018 13:00:00' for db : jj2              March 12, 2018 5:51:11 PM UTC       Failure
4b450741-f1b8-4e0c-a3fb-8a05b4f47dab     Create Longterm Backup with TAG-DBTLongterm1522439183710ORy for Db:jj2      March 30, 2018 7:51:48 PM UTC       Failure
[root@jj2 ~]# dbcli describe-job -i 4b450741-f1b8-4e0c-a3fb-8a05b4f47dab

Job details
----------------------------------------------------------------
                     ID:  4b450741-f1b8-4e0c-a3fb-8a05b4f47dab
            Description:  Create Longterm Backup with TAG-DBTLongterm1522439183710ORy for Db:jj2
                 Status:  Failure
                Created:  March 30, 2018 7:51:48 PM UTC
                Message:  DCS-10001:Internal error encountered: Wallet Status not found as OPEN-AUTOLOGIN/OPEN-UNKNOWN(UpgradeDb Case).

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
task:TaskZLockWrapper_243                March 30, 2018 7:51:51 PM UTC       March 30, 2018 7:52:03 PM UTC       Failure
task:TaskSequential_244                  March 30, 2018 7:51:51 PM UTC       March 30, 2018 7:52:03 PM UTC       Failure
Database container validation            March 30, 2018 7:51:51 PM UTC       March 30, 2018 7:51:51 PM UTC       Success
libopc existence check                   March 30, 2018 7:51:51 PM UTC       March 30, 2018 7:51:51 PM UTC       Success
Backup Validations                       March 30, 2018 7:51:51 PM UTC       March 30, 2018 7:52:03 PM UTC       Failure

In conclusion, troubleshooting the integrated Oracle DBCS Automatic Backup involves a systematic approach to identify and resolve issues affecting the backup process. Following the steps outlined above, you can effectively diagnose and address common problems encountered during backup operations.